Computer systems which accept data streams generated by operating a stylus are becoming commonplace. A stylus-based user interface generally comprises a pen (called a stylus) and a digitizing tablet. In many application programs, stylus-based user interfaces are superior to keyboards as a means for entering data. Such is the case, for instance, when a user of the computer has only one hand available for data entry. Other cases include, but are not limited to, those in which a keyboard would add too much bulk or weight to a data processing system designed to be highly portable or the case of a system designed for operation by a user who does not know how to or is physically unable to type.
Referring to FIG. 1, an application program 105 which receives a stylus-based input is called a stylus-based application programs (SBAPP) 105. The SBAPP 105 is designed and operated within a stylus-based system environment (SBENV) 100. The SBENV 100 comprises hardware and software components that are needed to capture, display and recognize handwriting input, and to display and if necessary correct results derived from incorrect recognition of the handwriting input. More specifically, the SBENV 100 comprises a pen 110, a digitizing tablet 120, a display device 130 (which may be integrated with the digitizing tablet), an operating system 140, tablet and display device drivers 150, a "stylus-aware window system" 160, one or more handwriting recognizers 170, and a set of error correction procedures 180.
Compared to an input data stream from a keyboard or mouse, an input stream from a stylus-based user interface is more difficult for the system to interpret and makes the development of stylus-based application programs very complex. The input stream of a keyboard or mouse (generally) unambiguously reflects a user's intention, that is, to select a particular keyboard key or mouse button. The application program may or may not be able to respond meaningfully to that particular input data, but the input data itself is clear. The SBENV 100, on the other hand, functions as a source of both character data (such as text, function keys and editing commands) and gesture data (i.e., mouse data such as pointing and selecting).
The input data stream of a stylus-based user interface consists of a sequence of (x,y) coordinate pairs which describe the locus of the stylus as the user writes. A "stroke" is a subsequence of (x,y) pairs, delimited by "pen-down" and "pen-up" events. A pen-down event occurs when the stylus first touches the digitizing tablet. A pen-up event occurs when the stylus next leaves the digitizing tablet.
A sequence of strokes may define a letter or a word. If so, the handwriting recognizer 170 receives the sequence of strokes and generates from it a sequence of codes (e.g., ASCII codes) as output for the recognized result.
Alternatively, a sequence of strokes may define a "gesture". In the context of handwriting recognition and error correction, a gesture is analogous to an editing command. For example, a horizontal line drawn over a displayed word may be interpreted as a request to delete that word and the space it occupies. An elliptical loop around a word may imply a request to edit that word, possibly causing it to appear in an "editing window". The distinction between one sequence of strokes being a gesture and another sequence being a letter or a word depends on such factors as the shape and size of the strokes and the context in which they arise.
If the sequence of strokes defines a gesture, the handwriting recognizer 170 receives the sequence and generates from it a structure which comprises a gesture code (to identify the gesture, e.g., a horizontal line) and an ordered set of points, referred to as "semantic points" or "hot points", which are used in the interpretation of the gesture as a command (the leftmost and rightmost points of a horizontal line determine the domain of the command, i.e., which letters and how much space will be removed).
The stylus-aware window system 160 is a conventional window system (one which relies on keyboard and mouse for user interface interactions, e.g., X Windows, 05/2 PM, Microsoft windows) which has been enhanced to support a stylus through mouse emulation and special "writing windows" in which a user can write and draw. Mouse emulation is used in all non-writing windows. For example, a program in a menu window may be "selected" by tapping the stylus once over the displayed program name; tapping once more may activate the selected program. Writing and drawing within a writing window produces stroke sequences which need to be processed by the handwriting recognizer 170.
The handwriting recognizer (HWR) 170 translates the user's natural writing (captured as a sequence of strokes) into a corresponding code set (e.g., ASCII). The handwriting recognizer 170 may be initiated to perform recognition when some event occurs. For example, the user changes from one writing window to another; or there is a timeout on new writing, i.e., no stroke has been drawn by the user for a sufficiently long time. Alternatively, recognition may be constantly active in the background. This allows the stylus-based system to support concurrently writing new strokes, recognizing received completed strokes, and displaying recognition results. As the user writes and generates strokes, they are sent to the application program 105 which passes them to the handwriting recognizer 170.
The handwriting recognizer 170 interprets these strokes using spatial and temporal considerations, generates a response such as "No Result Yet", "Result Available", or "Don't Know", and sends that response, packaged within a structure, to the application program 105. While all these computations are occurring, the user may be writing and generating new strokes. These strokes will be received by the application program 105. The application program 105, in turn, sends the strokes to the handwriting recognizer 170. The handwriting recognizer 170 then attempts to interpret the strokes.
Occasionally handwriting recognizers produce incorrect results. When this happens the user needs to edit the incorrect symbols using stylus-based editing procedures (as he or she would do in a keyboard-mouse environment by using keyboard-mouse editing procedures).
Ordinarily, error corrections are initiated by the user through stylus-based protocols. These protocols will be referred to as "error correction procedures" (ECPs) 180. Each ECP 180 entails a well defined set of stylus-based dialogs through which the user completes desired error corrections. Some ECPs which the user can employ involve:
Selection from a list of possible alternative code sequences which may be associated readily with the particular code sequence that is targeted for correction. The alternatives may be produced either by the handwriting recognizer or may be assembled by the user through observation of handwriting recognizer errors; PA1 rewriting the incorrectly recognized code sequence; the use of "gestures", which are interpreted as traditional editing commands like insert, delete, replace, etc.; For example, to replace two characters, the user might draw a line through the characters and handwrite the replacement characters. To insert a missing character, the user might make a caret where the character is to be inserted, and handwrite the character; PA1 the use of code menus, e.g., a keyboard displayed on the screen; or PA1 use of a combination of methods, such as marking a character to be replaced with a line gesture and typing the replacement character with a displayed keyboard.
A user may activate more than one ECP 180 to complete correction of some recognizer output. The application program 105 first receives the result from the handwriting recognizer 170. The user initiates error correction after the result is displayed via display device 130 by the application program 170.
There may be cases where the handwriting recognizer 170 is not able to produce a preferred interpretation for a sequence of strokes. The handwriting recognizer 170 may produce several likely possibilities. In such cases, multiple results may be presented to the user, thus requiring the user to select from a set of possible interpretations, or force a specific interpretation. This process is termed "weak recognition correction." It differs from regular error correction in that the need for user editing is established (by the handwriting recognizer 170) before the result is given to the application program 105.
In the SBENV 100 just discussed, the application program 105 deals directly with the handwriting recognizer 170, in fact with all recognizers which it may be using. This causes considerable complexity in the application program 105 since handwriting recognizers 170 are complex functional packages with correspondingly complex usage interfaces. Moreover, changes in the handwriting recognizer's 170 application programming interface (not shown) will usually require changes within the application program 105.
Moreover, application developers create specific error correction procedures for each application program. Stylus-based application development environments, such as Windows for Pen Computing (Microsoft Corp.) and PenPoint (Go Corp.), provide user interface procedures for pen entry, for pen activated buttons and menus, and for pen operated simulated keyboards, but do not provide procedures for error correction.
There are a small number of stylus-based, user-driven error correction paradigms, and it would be desirable to have these implemented as a code library. However, they must be independent of the application presentation processes, as these are application specific. Unfortunately, many of the error correction paradigms require the user to identify the erroneous results by pointing at them on the display. Thus, the error correction library procedures and the application must cooperate through an application independent protocol. Also, the error correction procedures may need to receive, recognize, and process strokes while they are active. These strokes will be diverted from the application.
The burden of complexity and maintenance on the application program 105 due to support of the handwriting recognizer's 170 external interface (not shown) increases if the ECPs 180 are integrated within the application program 105. This occurs because some ECPs 180 use the handwriting recognizer 105 and, therefore, they are affected by changes in the handwriting recognizer's 170 external interface, directly or indirectly.
The difficulty in writing application programs given the above problems has limited the marketability of SBENVs. Therefore, what is needed is a system that allows application programs that rely on a stylus-input to be designed without dependencies on the specific interfaces used by the stylus-input handwriting recognizers and the error correction modules.